Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added s3_storage backend (WIP) #37

Merged
merged 1 commit into from
Mar 21, 2024
Merged

Conversation

percona-ysorokin
Copy link
Collaborator

@percona-ysorokin percona-ysorokin commented Mar 19, 2024

Reworked how code is checked out in GitHub Workflows.
Inside '${{github.workspace}}' we now have:

  • 'src' (for the main project source tree)
  • 'aws-sdk-cpp' (for AWS SDK C++ source tree with submodules)

All Github Workflows 'checkout' actions are now instructed to fetch tags as well
('fetch-tags: true'). For our own code we also fetch full commit history
('fetch-depth: 0').

Reworked the way how we form labels for matrix jobs:
'<BUILD_TYPE>-<COMPILER_LABEL>', where
'<BUILD_TYPE>' is either 'Debug', 'RelWithDebInfo', or 'ASan' and
'<COMPILER_LABEL>' is either 'clang17' or 'gcc13'.
These labels are used in directory names and in cache keys.

Changed the way how Boost libraries are cached: instead of caching the binary
tarball ('.tar.bz2'), the unpacked directory is cached. This helps to eliminate
compressing already compressed data. Caching key now also includes the library
version (currently, 'boost-libraries-1-83-0').

Added a new step that shows '${{github.workspace}}' and '${{runner.temp}}' for
diagnostic purposes.

Added a new step that shows the content of the following directories:

  • '${{github.workspace}}'
  • '${{runner.temp}}'
  • '${{runner.temp}}/deps'
    for diagnostic purposes.

GitHub Workflows now builds AWS SDK C++ libraries.

  • Currently the selected version is 1.11.286 but it is configurable via
    ${{env.AWS_SDK_CPP_MAJOR}}, ${{env.AWS_SDK_CPP_MINOR}},
    and ${{env.AWS_SDK_CPP_PATCH}} variables.
  • The code is checked out from the 'aws/aws-sdk-cpp' GitHub repo (recursively
    with submodules) by '1.11.286' tag.
  • CMake is instructed to build only static versions of the libraries
    with all the dependencies ('-DBUILD_SHARED_LIBS=OFF' and
    '-DFORCE_SHARED_CRT=OFF').
  • The same compiler and build configuration as for the main project are used to
    build the libraries ('-DCMAKE_BUILD_TYPE=...', '-DCMAKE_C_COMPILER=...', and
    '-DCMAKE_CXX_COMPILER=...').
  • C++ standard is set to 'c++20' as in the main project ('-DCPP_STANDARD=20').
  • In order to instruct AWS SDK C++ to use 'libc++' standard library
    implementation (for 'clang' compiler), '-DCMAKE_CXX_FLAGS_INIT=-stdlib=libc++'
    option is passed manually for such configurations.
  • For Address Sanitizer builds '-DENABLE_ADDRESS_SANITIZER=ON' option is added.
  • Unity build is enabled ('-DENABLE_UNITY_BUILD=ON').
  • Building/running tests is disabled ('-DENABLE_TESTING=OFF' and
    '-DAUTORUN_UNIT_TESTS=OFF').
  • Currently only 's3-crt' component is build ('-DBUILD_ONLY=s3-crt').
  • The full specification of these options is available at
    https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/cmake-params.html
  • 'conan' package manager recipe
    https://github.com/conan-io/conan-center-index/blob/master/recipes/aws-sdk-cpp/all/conanfile.py
    and 'vcpkg' port
    https://github.com/microsoft/vcpkg/blob/master/ports/aws-sdk-cpp/portfile.cmake
    were taken as examples.

The libraries are built in the
'${{github.workspace}}/aws-sdk-cpp-build-${{matrix.config.label}}' directory.
The libraries are installed into the
'${{runner.temp}}/deps/aws-sdk-cpp-install-${{matrix.config.label}}' directory.
The build results (the content of the installation directory) are cached under
the 'ws-cpp-sdk-libraries-<SDK_VERSION>-<BUILD_TYPE>-<COMPILER_LABEL>' key
(e,g, 'aws-cpp-sdk-libraries-1-11-286-RelWithDebInfo-clang17'). This allows to
skip building AWS SDK C++ libraries every time a new PR is created. The rebuild
is expected to happen only when the SDK version is updated or a new version of
the compiler is added.

'-DCPP_STANDARD=20' CMake configuration option is now also added for the main
project to synchronize with AWS SDK C++ libraries.

Main 'CMakeLists.txt' file now tries to find AWS SDK C++ libraries via
'find_package()'. Developers are now expected to pass
'-DCMAKE_PREFIX_PATH=<AWS_SDK_CPP_INSTALL_DIR>' CMake option when they configure
the project, where '<AWS_SDK_CPP_INSTALL_DIR>' should point to the AWS SDK C++
installation directory.

Main application target now depends on 'aws-cpp-sdk-s3-crt'.

Added a stub for the second implementation of the
'binsrv::basic_storage_backend' interface, called 'binsrv::s3_storage_backend',
which in future will provide basic operations for working with AWS S3 storage.

'binsrv::storage_backend_factory' factory class extended with a new concrete
implemetation. Now:

  • for 'fs' we return an instance of 'binsrv::filesystem_storage_backend'
  • for 's3' we return an instance of 'binsrv::s3_storage_backend'.

'binsrv::basic_storage_backend' interface extended with one more abstract method
'get_description()':

  • 'binsrv::filesystem_storage_backend' ('fs') implementation returns
    "local filesystem"
  • 'binsrv::s3_storage_backend' ('s3') implementation returns
    "AWS S3 (SDK <SDK_VERSION>)" ('<SDK_VERSION>' is populated from the
    'Aws::SDKOptions' struct initialized via 'Aws::InitAPI()' call).

Main application extended with writing 'get_description()' output into the log.

@percona-ysorokin percona-ysorokin force-pushed the s3_storage_backend branch 15 times, most recently from 577e252 to 4dca5ef Compare March 21, 2024 01:03
…orkflows

Reworked how code is checked out in GitHub Workflows.
Inside '${{github.workspace}}' we now have:
- 'src' (for the main project source tree)
- 'aws-sdk-cpp' (for AWS SDK C++ source tree with submodules)

All Github Workflows 'checkout' actions are now instructed to fetch tags as well
('fetch-tags: true'). For our own code we also fetch full commit history
('fetch-depth: 0').

Reworked the way how we form labels for matrix jobs:
'<BUILD_TYPE>-<COMPILER_LABEL>', where
'<BUILD_TYPE>' is either 'Debug', 'RelWithDebInfo', or 'ASan' and
'<COMPILER_LABEL>' is either 'clang17' or 'gcc13'.
These labels are used in directory names and in cache keys.

Changed the way how Boost libraries are cached: instead of caching the binary
tarball ('.tar.bz2'), the unpacked directory is cached. This helps to eliminate
compressing already compressed data. Caching key now also includes the library
version (currently, 'boost-libraries-1-83-0').

Added a new step that shows '${{github.workspace}}' and '${{runner.temp}}' for
diagnostic purposes.

Added a new step that shows the content of the following directories:
- '${{github.workspace}}'
- '${{runner.temp}}'
- '${{runner.temp}}/deps'
for diagnostic purposes.

GitHub Workflows now builds AWS SDK C++ libraries.
* Currently the selected version is 1.11.286 but it is configurable via
  ${{env.AWS_SDK_CPP_MAJOR}}, ${{env.AWS_SDK_CPP_MINOR}},
  and ${{env.AWS_SDK_CPP_PATCH}} variables.
* The code is checked out from the 'aws/aws-sdk-cpp' GitHub repo (recursively
  with submodules) by '1.11.286' tag.
* CMake is instructed to build only static versions of the libraries
  with all the dependencies ('-DBUILD_SHARED_LIBS=OFF' and
  '-DFORCE_SHARED_CRT=OFF').
* The same compiler and build configuration as for the main project are used to
  build the libraries ('-DCMAKE_BUILD_TYPE=...', '-DCMAKE_C_COMPILER=...', and
  '-DCMAKE_CXX_COMPILER=...').
* C++ standard is set to 'c++20' as in the main project ('-DCPP_STANDARD=20').
* In order to instruct AWS SDK C++ to use 'libc++' standard library
  implementation (for 'clang' compiler), '-DCMAKE_CXX_FLAGS_INIT=-stdlib=libc++'
  option is passed manually for such configurations.
* For Address Sanitizer builds '-DENABLE_ADDRESS_SANITIZER=ON' option is added.
* Unity build is enabled ('-DENABLE_UNITY_BUILD=ON').
* Building/running tests is disabled ('-DENABLE_TESTING=OFF' and
  '-DAUTORUN_UNIT_TESTS=OFF').
* Currently only 's3-crt' component is build ('-DBUILD_ONLY=s3-crt').
* The full specification of these options is available at
  https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/cmake-params.html
* 'conan' package manager recipe
  https://github.com/conan-io/conan-center-index/blob/master/recipes/aws-sdk-cpp/all/conanfile.py
  and 'vcpkg' port
  https://github.com/microsoft/vcpkg/blob/master/ports/aws-sdk-cpp/portfile.cmake
  were taken as examples.

The libraries are built in the
'${{github.workspace}}/aws-sdk-cpp-build-${{matrix.config.label}}' directory.
The libraries are installed into the
'${{runner.temp}}/deps/aws-sdk-cpp-install-${{matrix.config.label}}' directory.
The build results (the content of the installation directory) are cached under
the 'ws-cpp-sdk-libraries-<SDK_VERSION>-<BUILD_TYPE>-<COMPILER_LABEL>' key
(e,g, 'aws-cpp-sdk-libraries-1-11-286-RelWithDebInfo-clang17'). This allows to
skip building AWS SDK C++ libraries every time a new PR is created. The rebuild
is expected to happen only when the SDK version is updated or a new version of
the compiler is added.

'-DCPP_STANDARD=20' CMake configuration option is now also added for the main
project to synchronize with AWS SDK C++ libraries.

Main 'CMakeLists.txt' file now tries to find AWS SDK C++ libraries via
'find_package()'. Developers are now expected to pass
'-DCMAKE_PREFIX_PATH=<AWS_SDK_CPP_INSTALL_DIR>' CMake option when they configure
the project, where '<AWS_SDK_CPP_INSTALL_DIR>' should point to the AWS SDK C++
installation directory.

Main application target now depends on 'aws-cpp-sdk-s3-crt'.

Added a stub for the second implementation of the
'binsrv::basic_storage_backend' interface, called 'binsrv::s3_storage_backend',
which in future will provide basic operations for working with AWS S3 storage.

'binsrv::storage_backend_factory' factory class extended with a new concrete
implemetation. Now:
- for 'fs' we return an instance of 'binsrv::filesystem_storage_backend'
- for 's3' we return an instance of 'binsrv::s3_storage_backend'.

'binsrv::basic_storage_backend' interface extended with one more abstract method
'get_description()':
- 'binsrv::filesystem_storage_backend' ('fs') implementation returns
  "local filesystem"
- 'binsrv::s3_storage_backend' ('s3') implementation returns
  "AWS S3 (SDK <SDK_VERSION>)" ('<SDK_VERSION>' is populated from the
  'Aws::SDKOptions' struct initialized via 'Aws::InitAPI()' call).

Main application extended with writing 'get_description()' output into the log.
@percona-ysorokin percona-ysorokin merged commit d4bae0e into main Mar 21, 2024
7 checks passed
@percona-ysorokin percona-ysorokin deleted the s3_storage_backend branch March 21, 2024 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant